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UNITED STATES 
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WASHINGTON, D.C. 20555-0001 




OFFICE OF THE 
INSPECTOR GENERAL 



September 30, 2010 



MEMORANDUM TO: 



R. William Borchardt 

Executive Director for Operations 



FROM: 



Joseph A. McMillan /RA/ 
for Investigations 



SUBJECT: 



ESSENTIAL SERVICE WATER LEAK AT BYRON 
NUCLEAR STATION (OIG CASE NO. 08-12) 



This memorandum conveys the results of an Office of the Inspector General (OIG) 
investigation into a leak that occurred in the essential service water system (ESW), a 
significant safety system, at Byron Nuclear Station (Byron) on October 19, 2007. This 
leak necessitated a 12-day shutdown of both reactors located at that site. This was a 
significant event, and NRC initiated a Special Inspection Team (SIT) soon after the 
shutdown to evaluate licensee actions surrounding the ESW failure. The SIT found that 
the licensee had not taken timely corrective action to correct degradation in ESW piping, 
and that a licensee analysis of the degradation's impact was faulty. 

awareness of corrosion in the 



in the months just prior to the leak. OIG found that: 

• The NRC oversight of licensee operability decisionmaking was not successful in 
learning of the steady reduction in margin of wall thickness of the ESW piping 
over a 7-month period until just 2 days before pipe failure. 

• Although the Byron resident inspectors carried out routine oversight 
responsibilities in accordance with agency requirements, the licensee's failure to 
analyze a problem correctly resulted in the resident inspectors' lack of awareness 
of a significant problem with the ESW. 
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• Although NRC inspection guidance conveys an expectation that resident 
inspectors should be aware of the status of safety systems, it was not specific 
enough to result in effective oversight of the degraded ESW by the resident 
inspectors. 

This report addresses circumstances typically experienced by resident inspectors, who 
are expected to be aware of plant conditions but are dependent, to a certain extent, on 
the licensee's ability to identify and correct plant problems. The Byron ESW event 
illustrates some of the challenges of this dependency and offers insights for meeting 
similar challenges in the future. 

I. Byron ESW 

Byron, located in northern Illinois, has two 1 ,200-megawatt output Westinghouse 
pressurized water reactors. The power station is owned and operated by Exelon. Like 
other pressurized water reactors, Byron has a safety related heat removal system that is 
used to remove decay heat and cool critical components during accident conditions. At 
Byron, the ESW fulfills these safety functions. 

The ESW is common to both units at Byron and is physically a very large system. It 
includes a dedicated cooling tower that serves as the ultimate heat sink for the Byron 
reactors. While much of the ESW piping is above ground, the portion of the piping that 
transports water to and from the ESW cooling tower is underground. 

The ESW cooling tower consists of eight separated cells. Each cooling tower cell 
contains a fan and spray nozzles. The pipes that convey the heated ESW water into 
each cooling tower cell are 24 inches in diameter, made of carbon steel, and have a 
nominal pipe wall thickness of 0.375 inches. The pipes emerge from the ground 
through a concrete floor, which is located inside a concrete valve vault. After 
penetrating the vault floor, each pipe connects to a bolted flange. The pipe connected 
to the other end of the bolted flange rises vertically and exits the vault. 

Each pipe section above the flange includes a single valve. The pipes direct the ESW 
water onward to the spray nozzles in the cooling tower. The vertical section of ESW 
piping inside the vault, above and below the flange, is called a pipe riser, At Byron, 
these pipe risers are designated "OA" through "OH" and these designations correspond 
to the eight cooling towers that they supply. The October 19, 2007, leak that resulted in 
Byron's shutdown occurred in the "OC" pipe riser. 
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The eight valve vaults are made of concrete and each has a removable steel hatch 
plate that allows personnel entry into the vault, an area which is otherwise inaccessible. 
The hatch plates require a crane for removal. The valve vaults are designed to provide 
physical protection for their contents against the impact of high velocity debris 
associated with a tornado. 

II. Byron ESW Corrosion History 

Byron has a history of ESW piping corrosion in the cooling tower valve vaults. Byron 
staff as far back as 1990 had identified excessive corrosion on this piping. At that time, 
the licensee attributed excessive corrosion of the piping to a failure of the pipe coatings. 
The condition was described as extensive with loose scaly rust that originated on the 
outside surface of the piping. 

In 1994, the licensee initiated plans to replace the ESW carbon steel piping in the vaults 
with stainless steel piping because of the enhanced corrosion resistance of stainless 
steel. In 1997 and 1998, the ESW piping above the bolted flange was replaced; 
however, the short section of pipe riser coming through the concrete floor and 
terminating below the bolted flange was not replaced. While the lower section of the 
pipe was originally within the scope of the planned changes, Byron later removed it from 
the scope and replacement never occurred. 

Between 1998 and 2007, the Byron staff continued periodic visual inspections as 
required by the American Society of Mechanical Engineering code. Until 2006, most of 
the inspections noted no degradation, although some instances of corrosion were 
identified. In 2006, two pipe risers that were inspected were documented as having 
some corrosion, but the corrosion was not characterized as severe. 

III. ESW Corrosion in 2007 

Between March 2007 and October 2007, Byron staff conducted a series of inspections 
of the pipe riser, identified progressive thinning of the pipe wall had resulted in wall 
thickness that was less than the established standard, and four times reduced the 
minimum value for allowed riser pipe wall thickness: first from 0.375 to 0.153 inches, 
then from 0.153 to 0.121 inches, again from 0.121 to 0.06 inches, and finally from 0.06 
inches to 0.03 inches. The progression of inspections and decisions to reduce the wall 
thickness standard was as follows: 
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• March 2007: During a periodic visual inspection, the Byron staff noted significant 
corrosion on the "0C" pipe riser, including metal corrosion product 

flaking off the pipe. The staff determined that corrosion had worsened since the 
last inspection (May 2006), and determined that additional inspection was 
needed of other pipe risers. In May 2007, Byron staff performed a visual 
inspection of the "OA" pipe riser and found significant corrosion, in response to 
the corrosion identified on the "OA" and "0C" pipe risers, the staff decided to 
schedule ultrasonic testing of the "0E" riser to determine the thickness of that 
pipe. The Byron staff also established a new minimum pipe wall thickness of 
0.153 inches in expectation that future measurements of the pipe wall thickness 
would be less than the nominal value of 0.375 inches and that the pipe wall 
would have adequate thickness as long as it was greater than 0.153 inches. 

• June 4, 2007: Byron staff visually inspected the "OH" pipe riser and found that 
the protective coating that was supposed to be on the pipe was degraded and 
the pipe's bare metal was exposed. 

• June 14, 2007: Byron completed ultrasonic pipe wall thickness measurements 
for the ESW pipe riser in the "0E" valve vault. The testing identified localized 
areas below the allowed minimum wall thickness of 0.153 inches that had just 
been established in May. The two thinnest areas measured had wall thickness 
values of 0.122 and 0.124 inches, well below what had been established as the 
allowed minimum. 

• July 1 1 , 2007: After the below-threshold thicknesses were identified in the "0E" 
pipe riser, Byron engineering staff completed a detailed evaluation and lowered 
the minimum acceptable wall thickness for continued operability to 0.121 inches. 
The calculation that determined the new minimum standard was documented in 
an "engineering change" document. It was not unusual at Byron at that time to 
document what was essentially an operability determination in an engineering 
change document. The document also stated that action had been taken to 
present the pipe riser thinning issue to the Plant Health Committee and 
recommend that the piping be restored to original design conditions. 

• September 2007: Visual inspections were performed on the "0B" and "0D" pipe 
risers and no degradation was noted, although degradation had been noted on 
both of these risers in previous inspections and no repairs had been made. 
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• October 10, 2007: The "OH" pipe riser wall thickness was measured at 0.-085 
inches. Byron staff lowered the allowed thickness from 0.121 to 0.06 inches 
through a formal operability evaluation dated October 12, 2007. The 
documentation of operability in a formal operability evaluation rather than an 




engineering change document indicated that the licensee believed the issue was 
significant. 

• October 17, 2007: the "0B" pipe riser wall thickness was measured at 10 
locations; all measured thicknesses were less than 0.1 inches, including one 
location measuring 0.047 inches. The licensee subsequently lowered the 
allowed thickness to 0.03 inches. 

• October 19, 2007: While Byron staff were brushing corrosion product off the "0C" 
pipe riser to conduct further ultrasonic testing, the brush scraped through the 
pipe wall, the pipe began leaking, and the licensee was forced to shut down both 
Byron reactors to make repairs. 

IV. NRC Awareness of Corrosion Prior to Leak 

A. Corrective Action Program Documentation 

The degradation of the pipe risers as well as the licensee's actions to evaluate and 
correct the degradation was documented in Byron's Corrective Action Program (CAP). 
During 2007, about 13,300 issues were entered into the CAP. This equates to an 
average of 37 issues per day, although the daily rate can vary greatly based on 
conditions or activities (e.g., problems with a particular system, plant outage) underway 
at the plant. Between March 2007 (the point at which corrosion was identified on the 
"0C" pipe riser) and October 19, 2007 (date of the leak), there were about 277 issues 
that concerned some aspect of the ESW, and about 25 that dealt specifically with pipe 
riser corrosion or degradation. This included action requests in June and July to 
perform ultrasonic testing analyses and repairs of the pipe risers. 

NRC Inspection Procedure 71152, "Problem Identification and Resolution," establishes 
the expectations for resident inspector review of the CAP. The procedure instructs 
resident inspectors to review each CAP input and follow up on selected items and be 
alert for adverse performance trends. 
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The two NRC resident inspectors who worked at Byron during the months leading up to 
the leak both stated that they reviewed every CAP issue report that Byron generated. 
The senior resident inspector recalled thinking in September 2007 that there was an 
unusually high number of issue reports on the ESW piping degradation, perhaps as 
many as a dozen. However, he told OIG he did not follow up on this information prior to 
the leak because he believed the degradation was minor surface corrosion and that the 
licensee was addressing the issue appropriately. He also learned, through review of 
materials for a presentation at the September Byron Nuclear Safety Review Board 
meeting, the licensee had found the ESW piping was degraded and had performed 
operability determinations and decided to replace the pipe. 

B. Discussion During Licensee Meetings 

Commercial nuclear power plants have a structure of routine meetings to identify and 
resolve issues. For example, most nuclear power plants, including Byron, hold daily 
Plan of the Day meetings to coordinate work activities at the site each workday. 
Meetings are also held regularly to plan and perform maintenance, establish priorities 
for funding for repairing plant equipment, and provide management oversight of the 
plant activities and performance. 

During the summer of 2007, the Byron staff used this structure of meetings to address 
the problems that had been discovered with the ESW pipe riser degradation. For 
example, the corrosion found on the "OA" pipe riser in May was presented to the 
Management Review Committee, which reviews inputs to the CAP. The Management 



Review Committee assigned the issue to an engineering rapid response team to 
address the wall thinning. 

The Plant Health Committee, which helps establish the priority for plant repairs and 
modifications and meets weekly, identified the ESW riser piping as a new focus area on 
June 25, 2007. On July 16, the riser piping degradation ultrasonic testing results were 
presented at the committee meeting, and on August 16, another presentation on pipe 
riser degradation was held to seek approval of modifications to replace the degraded 
riser piping. On September 5, another presentation was held seeking approval to fund 
modifications. That presentation stated that the riser piping degradation had the 
potential to cause the loss of the plant ultimate heat sink due to a leak that could not be 
isolated and could compromise nuclear safety. In addition to these formal 
presentations, the plant system engineer was required to report on the system at these 
meetings at least biweekly throughout the summer of 2007. 



Byron also holds a daily Risk Meeting that includes Byron managers and discusses 
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current Issues and activities that could involve the site's after hours duty staff. If was 
while attending an October 17, 2007, Risk Meeting (2 days before the leak occurred) 
that a Byron resident inspector learned, for the first time, of the significance of the ESW 
piping degradation. 

Inspection Manual Chapter 2515, "Light -Water Reactor Inspection Program - 
Operations Phase," Appendix D, provides guidance to NRC inspectors on methods to 
maintain awareness of plant status. This chapter states that inspectors should attend 
licensee meetings on a routine basis to provide them with knowledge of overall plant 
status, but leaves it to the inspectors' discretion to select which specific meetings to 
attend. 

The Byron resident inspectors told OIG that they did attend Byron plant meetings on a 
regular basis. The senior resident inspector said the resident inspectors attended the 
Plan of the Day meeting about three times per week, and the daily Risk Meeting, about 
once a week. The inspectors also attended the operations shift turnover meeting on a 
daily basis and the Plant Health Committee meeting once every 3 or 4 weeks. Byron 
staff confirmed that the resident inspectors attended the meetings on a regular basis. 

The senior resident inspector said that after he learned about the extent of the 
degradation at the October 17, 2007, Risk Meeting, he e-mailed his regional manager to 
inform him about this issue. The regional manager recalled learning about the issue 
approximately 2 days before the leak. The regional manager said that when he learned 
of the pipe riser degradation, he and the regional engineering staff began engaging in 
telephone conversations with the licensee to understand the scope of the issue. 

C. Oversight of Qperabilitv Decisionmaking 

NRC Inspection Procedure Attachment 71 11 1.1 5,"Operability Evaluations," requires 
resident inspectors to inspect 19 to 25 operability evaluations each year as part of 
NRC's baseline inspection program. The procedure requires a proportion of the total 
sample to be reviewed each quarter, but contains no other timeliness requirements. 

The Byron senior resident inspector provided OIG with a list of 21 operability decisions 
the resident inspectors reviewed in 2007. None of the decisions reviewed pertained to 
the ESW pipe wall thinning; however, the sample included a portion of the nine formal 
operability evaluations performed at Byron in 2007, and some of the less formal 
operability determinations. The senior resident inspector stated that the sample 
selection was based on his judgment about the safety significance of each issue. 



The senior resident inspector also stated that while Byron completed a formal 
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operabLty^valuation of the "OH" pipe riser degradation on October 12, 2007, he'aid not 
review it before the leak, which occurred just 1 week later, on October 19, 2007. 

OIG learned that the SIT reviewed Byron's July 11, 2007, evaluation of the "0E" pipe 
riser that lowered the minimum acceptable wall thickness for continued operability from 
0.153 inches to 0.121 inches. The SIT determined that the calculation that arrived at 
the minimum wall thickness did not include all the loads to which the pipe was subject. 
The SIT concluded that the minimum acceptable wall thickness would have been 
greater than 0.121 inches had those loads been included in the calculation. OIG noted 
that this would bring the operability decision made by the Byron staff at the time into 
question since the measured wall thickness of 0.122 inches allowed little margin for 
calculation error. The SIT also noted problems with Byron's calculations to reduce the 
minimum acceptable wall thickness to 0.06 inches and, later, to 0.03 inches. OIG noted 
that while the SIT challenged the calculations, it did not question the overall reduction in 
minimum acceptable wall thickness during this time period. 

D. Resident Inspector Communication with Byron Staff 

OIG interviewed members of the Byron staff familiar with the event to determine the 
level of importance the Byron staff placed on the ESW degradation and the 
communication of this issue to NRC. The senior managers at the site were aware of the 
degradation, but did not recall being aware of its severity or extent until shortly before 
the leak occurred. None of the senior managers interviewed recalled the basis for 
considering the ESW operable in July, given the 0.1 21 -inch thickness standard in an 
actively corroding system with measured thickness as low as 0.122. An engineering 
manager familiar with the event and the root cause analysis performed by the licensee 
said that the Byron staff did not understand that the corrosion was active and the 
significance of that on operability until the leak occurred in October 2007. 

The Byron senior managers interviewed who maintained frequent communications with 
the NRC resident inspectors said they did not recall informing the resident inspectors 
about the problems during the months leading up to the ESW leak. 

Byron, like many commercial nuclear power plants, rates the overall condition of each 
plant system using a color-coded system, which is reported in a System Health Report 
and updated quarterly. The ESW system was rated as red (the worst condition rating 
for a system) from the 4 th quarter in 2006 through the 3 rd quarter in 2007. The senior 
resident inspector stated he was aware that the ESW system health report was red, 
indicating that Byron personnel viewed the system as having problems. However, 



neither the senior resident inspector nor the Byron system engineer responsible for the 
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ESW recall either party raising this topic or pipe riser corrosion with the other. 



The senior resident inspector said he believed the licensee personnel never mentioned 
the ESW pipe riser degradation to him because they had presumed it was not a 
problem. According to the senior resident inspector, the licensee would not raise an 
issue unless they viewed it as a problem. 

E. Walkdowns of Inaccessible Areas by Resident Inspectors 

As previously noted in section I (Byron ESW) of this report, the ESW pipe risers at 
Byron are located in normally inaccessible concrete valve vaults that require a crane for 
access. The senior resident inspector stated that although the Byron resident 
inspectors looked at many infrequently accessed areas, he was unaware of any NRC 
personnel entering any of the valve vaults prior to the October 2007 ESW leak. He 
stated that during the time leading up to the leak, there was no NRC requirement to 
periodically enter spaces that were normally inaccessible. He also said he was 
unaware that there was ESW piping in the valve vaults. A former Byron resident 
inspector noted that any licensee work inside the valve vaults required a crane, which 
should have increased the visibility of licensee entries into the vaults and the likelihood 
of the resident inspectors' awareness of the opportunity for vault access. 



leak, Appendix D instructed inspectors to tour a portion of accessible plant areas 
weekly, and noted that inspectors may increase the scope and frequency of plant tours 
during changing plant conditions, such as outages, and that plant tours could include 
areas that become accessible during outages. In response to the Byron leak event, 
NRC revised the Appendix D guidance in September 2008 to be more comprehensive 
and detailed with regard to inspection of normally inaccessible areas. 



V. Management Expectations for Resident Inspector Awareness of Degraded 
Safety Systems 

NRC senior regional managers told OIG that they expected resident inspectors to have 
knowledge of plant status and health and to have knowledge of degraded safety 
systems. One manager noted that there are different levels of degradation and that 
inspectors should have a general overview of degraded safety systems. Another 
manager stated that the resident inspectors should be aware of what safety systems are 
degraded. 



A recently retired Office of Nuclear Reactor Regulation manager told OIG that resident 
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inspectors should have knowledge of degraded safety systems such as the Byron ESW. 
He said that NRC's expectations for this should be clear and detailed, and should 
require inspectors to be proactive in looking for signs of degradation in significant safety 
systems. He said that good guidance for consideration of degraded safety systems 
could include degradation that causes the nuclear plant operators to enter into a shorter 
term Limiting Condition for Operation, operability evaluations on safety systems, and 
degradation that requires engineering analysis to resolve. He expected inspectors at 
every plant to know what safety systems are degraded. He also said that in situations 
where inspectors have not sampled or have not asked the right question, and licensees 
have not communicated, then NRC may not learn of a significant safety issue such as the 
degraded ESW piping at Byron. He stated, looking back at the event, that he would have 
expected the licensee to have had an information exchange with the resident inspectors 
about the ESW degradation. He stated that resident inspectors do not run the nuclear 
power plants they oversee, and they have to get information from the licensee to some 
degree. 

The senior resident inspector told OIG he understands he is expected to be aware of 
plant status and general system conditions, including degraded safety systems, but 
these expectations are vague, particularly with regard to level of detail of knowledge 
that is required. He stated that expectations for resident inspector knowledge of 
degraded safety systems are not specifically described in any of NRC's inspection 
guidance and that a working definition of "degraded" would be needed to be able to 
meet expectations to be aware of degraded safety systems. 

OIG reviewed NRC inspection guidance for specific guidance to inspectors on the 

Manual Chapter 2515, Appendix D, "Plant Status" which provides the most direct 
guidance for resident inspectors on monitoring of plant status and is the most likely 
place to find direction to resident inspectors to be knowledgeable of degraded safety 
systems. Appendix D states that resident inspectors should be "aware of emergent plant 
issues, potential adverse trends, current equipment problems, and ongoing activities, 
including their impact on plant risk." It also states that inspectors should look for 
obvious signs of degraded material condition and includes some detailed guidance such 
as checking for loose anchor bolts and substantial corrosion. However, Appendix D 
does not define "degraded" and does not charge inspectors with the responsibility of 
being aware of all safety systems that are degraded. 
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VI. Conclusion 

Although the Byron resident inspectors reviewed CAP entries, attended plant meetings, 
and inspected Byron operability decisionmaking in accordance with agency guidance, 
this level of oversight did not lead to knowledge of ongoing and significant ESW 
degradation over a 7-month period. The senior resident inspector only learned of the 
significance of the problem 2 days before the leak that caused the inoperability of a 
significant safety system and forced both reactors to be shut down. A major factor in 
the resident inspectors' lack of awareness of this issue was the licensee's poor handling 
of operability decisionmaking and the licensee's apparent lack of recognition of the 
significance of the problem. 

Please respond to this office within 120 days on what, if any, action you intend to take in 
response to this report. If you have any questions, please contact me, at 301-415-5929, 
or Rossana Raspa, Senior Level Assistant for Investigative Operations, at 
301-415-5925. 
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